Method and apparatus for composition of subtitles

ABSTRACT

Embodiments of the invention include a subtitling format encompassing elements of enhanced syntax and semantic to provide improved animation capabilities. The disclosed elements improve subtitle performance without stressing the available subtitle bitrate. This will become essential for authoring content of high-end HDTV subtitles in pre-recorded format, which can be broadcast or stored on high capacity optical media, e.g. the Blue-ray Disc. Embodiments of the invention include abilities for improved authoring possibilities for the content production to animate subtitles. For subtitles that are separate from AV material, a method includes using one or more superimposed subtitle layers, and displaying only a selected part of the transferred subtitles at a time. Further, colors of a selected part of the displayed subtitles may be modified, e.g. highlighted.

This application is a Continuation of co-pending U.S. application Ser.No. 12/800,418, filed May 14, 2010, herein incorporated by reference inits entirety, which is a Continuation-In-Part of U.S. application Ser.No. 10/535,106, filed May 16, 2005, now U.S. Pat. No. 7,852,411 hereinincorporated by reference in its entirety.

CROSS RELATED APPLICATIONS

This application claims benefit of U.S. application Ser. No. 12/800,418,filed May 14, 2010, which claims the benefit, under 35 U.S.C. §365 ofInternational Application PCT/EP03/12261, filed Nov. 3, 2003, which waspublished in accordance with PCT Article 21(2) on Jun. 3, 2004 inEnglish and which claims the benefit of European patent application No.02025474.4, filed Nov. 15, 2002.

FIELD OF THE INVENTION

The invention relates to a method and to an apparatus for composition ofsubtitles for audio/video presentations, which can be used e.g. for HDTVsubtitles in pre-recorded formats like the so-called Blue-ray Disc.

BACKGROUND OF THE INVENTION

The technique of subtitling for Audio-Visual (AV) material has been usedbeginning with the first celluloid cinema movies and further until therecent digital media appeared. The main target of subtitling has beenthe support of handicapped people or small ethnographic language groups.Therefore subtitling often aims at the presentation of text informationeven when having been encoded as graphic data like pixel maps. Thereforepre-produced AV material for broadcasting (Closed Caption, Teletext,DVB-Subtitle etc.) and movie discs (DVD Sub-Picture etc.) primarily areoptimized for subtitles representing simple static textual information.However, progress in PC software development for presentation andanimation of textual information induces a corresponding demand forpossibilities and features within the digital subtitling technique usedfor pre-recording and broadcasting. Using straightforward approacheswithout any special precautions, these increased requirements forsubtitling would consume a too big portion of the limited overallbandwidth. The conflicting requirements for a ‘full feature’ subtitleencompassing karaoke all through genuine animations are on one hand thecoding efficiency and on the other hand the full control for anysubtitle author.

For today's state of the art of digitally subtitling AV material withseparate subtitling information two main approaches exist: Subtitlingcan be based on either pixel data or on character data. In both cases,subtitling schemes comprise a general framework, which for instancedeals with the synchronization of subtitling elements along the AV timeaxis.

Character Data Based Subtitling:

In the character-based subtitling approach, e.g. in the teletext systemETS 300 706 of European analog or digital TV, strings are described bysequences of letter codes, e.g. ASCII or UNICODE, which intrinsicallyallows for a very efficient encoding. But from character strings alone,subtitling cannot be converted into a graphical representation to beoverlaid over video. For this, the intended character set, font and somefont parameters, most notably the font size, must either be codedexplicitly within the subtitling bitstream or an implicit assumptionmust be made about them within a suitably defined subtitling context.Also, any subtitling in this approach is confined to what can beexpressed with the letters and symbols of the specific font(s) in use.The DVB Subtitling specification ETS 300 743, in its mode of “characterobjects”, constitutes another state-of-the-art example ofcharacter-based subtitling.

Pixel Data Based Subtitling:

In the pixel-based subtitling approach, subtitling frames are conveyeddirectly in the form of graphical representations by describing them as(typically rectangular) regions of pixel values on the AV screen.Whenever anything is meant to be visible in the subtitling planesuperimposed onto video, its pixel values must be encoded and providedin the subtitling bitstream, together with appropriate synchronizationinfo, and hence for the full feature animation of subtitles all pixelchanged must be transported. Obviously, when removing any limitationsinherent with full feature animations of teletext, the pixel-basedapproach carries the penalty of a considerably increased bandwidth forthe subtitling data. Examples of pixel-based subtitling schemes can befound in DVD's sub-picture concept “DVD Specification for Read-Onlydisc”, Part 3: Video, as well as in the “pixel object” concept of DVBSubtitling, specified in ETS 300 743.

SUMMARY OF THE INVENTION

Embodiments of the invention include a subtitling format encompassingelements of enhanced syntax and semantic to provide improved animationcapabilities. The disclosed embodiments improve subtitle performancewithout stressing the available subtitle bitrate. This will becomeessential for authoring content of high-end HDTV subtitles inpre-recorded format, which can be broadcast or pressed on high capacityoptical media, e.g. the Blue-ray Disc. The invention includes abilitiesfor improved authoring possibilities for the content production toanimate subtitles.

Introduced by the disclosure are elements of syntax and semanticdescribing the color change for parts of graphics to display. This canbe used for highlight effects in applications like for example karaoke,avoiding the repeated transfer of pixel data.

Other disclosed elements of syntax and semantic facilitate the abilityof cropping parts of the subtitles before displaying them. By using thetechnique of subsequently transferred cropping parameters for an objectto display, a bit saving animation of subtitles becomes available. Suchcropping parameter can be used for example to generate text changes bywiping boxes, blinds, scrolling, wipes, checker boxes, etc.

Furthermore the disclosed elements can be used to provide interactivityon textual and graphical information. Especially the positioning and/orcolor settings of subtitles can be manipulated based upon user request.

DRAWINGS

Exemplary embodiments of the invention are described with reference tothe accompanying drawings and tables, which show:

FIG. 1: segment_type values for enhanced PCS and RCS;

FIG. 2: Enhanced page composition segment;

FIG. 3: Enhanced region composition segment;

FIG. 4: Example for the definition of a subtitle region and its locationwithin a page;

FIG. 5: Example for definition of a region sub-CLUT and region cropping;

FIG. 6: Resulting display example;

FIG. 7: Interactive usage of subtitles;

FIG. 8: Video and Graphics Planes;

FIG. 9: Video and Graphics Mixing and Switching.

EXEMPLARY EMBODIMENTS OF THE INVENTION

The invention can preferably be embodied based on the syntax andsemantic of the DVB subtitle specification (DVB-ST). To provide improvedcapabilities for the manipulation of graphic subtitle elements, thesemantics of DVB-ST's page composition segment (PCS) and regioncomposition segment (RCS) are expanded.

DVB_ST uses page composition segments (PCS) to describe the positions ofone or more rectangular regions on the display screen. The regioncomposition segments (RCS) are used to define the size of any suchrectangular area and identifies the color-lookup-table (CLUT) usedwithin.

Embodiments of the proposed invention keeps backward compatibility withDVB-ST by using different segment_types for the enhanced PCS and RCSelements, as listed in FIG. 1 showing segment type values according toDVB-ST, with additional values for enhanced PCS and enhanced RCS. Itwould also be possible to choose other values instead. Another approachfor keeping backward compatibility would be to keep the existingsegment_types and increase the version_number of the specification, e.g.by incrementing the subtitle_stream_id in the PES_data_field structure.

FIG. 2 shows the data structure of an enhanced page composition segment(PCS), containing a region_cropping section and a region_sub_CLUTsection. FIG. 3 shows the data structure of an enhanced regioncomposition segment (RCS), containing an identifier sub_CLUT_id for asub-color-look-up-table. With respect to original DVB-ST, all structuresshown are expanded. In the tables the additional entries are lines 15-28in FIG. 2 and line 16 in FIG. 3.

The enhanced PCS shown in FIG. 2 carries optional information about theregion cropping and optional information about the region_sub-CLUT forevery region listed. The two values of region_cropping andregion_sub_CLUT indicate if such optional information is available forthe current region in process. Therefore cropping and sub-CLUT may bedefined separately for every region. While region_cropping is used as aflag, as indicated by “if region cropping==0×01”, the region_sub_CLUTshows the value how many sub-CLUT positions are described. This is doneto provide different alternatives within the stream. Alternativesub-CLUT positions can be used to define different menu button positionsfor the display screen. Only one of them—the first one as a default—isactive and the user can change the position to navigate through thedifferent predefined positions pressing the remote for example.

The enhanced RCS shown in FIG. 3 carries the sub_CLUT_id identifying thefamily of CLUTs that applies to this region. This is done to re-useCLUTs for different regions and different region sub_GLUTs as well. Theenhanced PCS and enhanced RCS elements provide the ability thatsubtitles can be manipulated independent from the encoding method i.e.independent from whether they are encoded as character data or pixeldata.

The enhanced PCS and RCS can be used to perform many different animationeffects for subtitles. Those could be wiping boxes, blinds, scrolling,wipes, checker boxes, etc. The following figures show an applicationexample for karaoke. FIG. 4 shows the definition of a region Rcontaining lyrics of a song displayed for karaoke. The letters of thesubtitle may be encoded as pixel data or as character data as well. Theregion_vertical_address RVA and the region_horizontal_address RHA definethe location of the subtitle within the frame, or page PG, to display.

FIG. 5 depicts in the upper part region cropping, and in the lower partthe location of the region sub-CLUT. Region cropping defines which partof the region is effectively displayed. This is achieved by fourparameters RHC,RVC,RCH,RCW indicating the start coordinates and the sizeof the fragment to display. Region_horizontal_cropping RHC specifies thehorizontal address of the top left pixel of this cropping,region_vertical_cropping RVC specifies the vertical address of the topline of this cropping, region_cropping_width RCW specifies thehorizontal length of this cropping, and region_cropping_height RCHspecifies the vertical length of this cropping, wherein cropping meansthat part of the subtitles that is visible on a display.

The region sub-CWT location shown in the lower part of FIG. 5 defineswhich part of the region has to be displayed using a color-look-up-table(CLUT) different from the region CLUT. This is achieved by fourparameters SCHA,SCVA,SCH,SCW indicating the start coordinates and thesize of the sub-region used by the sub-CLUT. All coordinate parametersare to be understood relative to the region the sub-CLUT belongs tosub_CLUT_horizontal_address SCHA specifies the horizontal address of thetop left pixel of this sub-CLUT, sub_CLUT_vertical_address SCVAspecifies the vertical address of the top line of this sub-CLUT,sub_CLUT_width SGW specifies the horizontal length of this sub-CLUT andsub_CLUT_height SCH specifies the vertical length of this sub-CLUT.

Picking up all parameters defined with the previous figures results inthe displayed subtitle as depicted in FIG. 6. The subtitle is notdepicted in whole on the display but only the cropped part of it.Furthermore the sub-CLUT was used to provide a highlight HT, so that theuser knows what to sing in the moment.

As the enhanced PCS are sent within MPEG packet elementary stream (PES)packets labeled by presentation time stamps (PTS), any effect can besynchronized to the AV.

Another idea of the invention is the superseding of subtitle animationparameters by the user. This offers a way to realize interactivesubtitles. The enhanced PCS parameters are transferred as a default, andthe user may change them via a remote control for example. Thus the useris able to move, crop or highlight the subtitle.

This could be an advantage for a user defined repositioning of asubtitling text, so that the user can subjectively minimize theannoyance by the subtitle text placement on top of the motion video.Also the color of the subtitles could be set according to userspreferences. FIG. 7 shows a block diagram for interactive subtitlemodifications. The default parameters DD read from a disc D aresuperseded by superseding data SD being generated upon the user actionUA and processed by a processor P.

Another application for overriding subtitle animation parameters likeposition, cropping rectangle, CLUTs and sub-CLUTs is the realization ofsome very basic sort of interactive gaming. The subtitle may carry pixeldata of an animated character. This character is subsequently moved onthe display screen driven by either user interaction, programmaticcontrol or both.

The overriding of subtitle animation parameters can be implemented in atleast two ways. The first option is that the overriding parameters SDreplace the parameters DD send in the bitstream. The second option isthat the overriding parameters SD are used as an offset that is added toor subtracted from the subtitle animation parameters DD send in thebitstream.

The enhanced PCS and RCS provide a lot more of animation capabilitiesnot explained. Following is a non-exhaustive list of examples: wipingboxes, blinds, scrolling, wipes, checker boxes in details.

Exemplary video and graphics planes are shown in FIG. 8 in an exemplary,schematic manner. A background is provided by either an MPEG-2 videolayer MVL or a still picture layer SPL. They are mutually exclusive,which means that not both of them need to be held in a buffer at a time.The next two layers comprise a subtitle layer SL and an AV sync typegraphics layer AVSGL. These two layers are in this exampleinterchangeable, meaning that either the subtitle layer SL or the AVsync type graphics layer AVSGL may have priority over the other. Thefront layer is a non-AV sync graphics layer NAVSGL, containing graphicsthat need not be synchronized with the AV content, such as e.g. menus orother on-screen displays. The inventive method can preferably be usedfor the subtitle layer SL, the AV sync graphics layer AVSGL and/or theNon-AV sync graphics layer NAVSGL.

FIG. 9 shows relevant components of an apparatus for video and graphicsmixing and switching. Data comprising either still picture data orMPEG-2 video data, further data for subtitles, data for animations anddata for non-AV sync graphics such as menu buttons, are retrieved from adisc D. Additionally or alternatively, data for subtitles, animationsand/or non-AV sync graphics can be received from a network NW, e.g.Internet. A processing unit CPU processes the non-AV sync graphics dataand sends the resulting data to a rendering device for non-AV syncgraphics RNAVG.

The apparatus contains a still picture decoder SPDec and an MPEG-2 videodecoder MVDec, but since only one of them is used at a time, a switch s1can select which data shall be used for further processing. Moreover,two identical decoders AVSGDec1,AVSGDec2 are used for decoding subtitleand animation data. The outputs of these two decoders AVSGDec1, AVSGDec2may be switched by independent switches s2,s3 to either a mixer MX, orfor preprocessing to a mixer and scaler MXS, which outputs its resultingdata to said mixer MX. These two units MX,MXS are used to perform thesuperimposing of its various input data, thus controlling the displayorder of the layers. The mixer MX has inputs for a front layer f2, amiddle front layer mf, a middle back layer mb and a background layer b2.The front layer f2 may be unused, if the corresponding switch s3 is in aposition to connect the second AV sync graphics decoder AVSGDec2 to themixer and scaler MXS. This unit MXS has inputs for front layer f1,middle layer m and background layer b. It superimposes these datacorrespondingly and sends the resulting picture data to the backgroundinput b2 of the mixer MX. Thus, these data represent e.g. a framecomprising up to three layers of picture and subtitles, which can bescaled and moved together within the final picture. The background inputb1 of the mixer and scaler MXS is connected to the switch s1 mentionedabove, so that the background can be generated from a still picture oran MPEG-2 video.

The output of the first AV sync graphics decoder AVSGDec1 is connectedto a second switch s2, which may switch it to the middle layer input mof the mixer and scaler MXS or to the middle back layer input mb of themixer MX. The output of the second AV sync graphics decoder AVSGDec2 isconnected to a third switch s3, which may switch it to the front layerinput f1 of the mixer and scaler MXS or to the middle front layer inputmf of the mixer MX.

Depending on the positions of the second and third switch s2,s3, eitherthe output of the first or the second AV sync graphics decoderAVSGDec1,AVSGD2 may have priority over the other, as described above.For having the data from the first decoder AVSGDec1 in the foreground,the second switch s2 may route the subtitle data to the middle backinput mb of the mixer MX, while the third switch s3 routes the animationgraphics data to the front input f1 of the mixer and scaler MXS, so thatit ends up at the background input b2 of the mixer MX. Otherwise, forhaving the data from the second decoder AVSGDec2 in the foreground, theswitches s2,s3 may route their outputs to the same unit, either themixer and scaler MXS or the mixer MX, as shown in FIG. 9.

1. A method for composition of subtitles, the method mixing andswitching video and graphics data, the data being read from a storagemedium or received from a network and comprising at least one of stillpicture data and MPEG video data, data for at least two layers ofsubtitles or animations, and data for non-synchronized graphics, themethod comprising the steps of: superimposing in a mixer video data of aback layer, at least two middle layers and a front layer; superimposingin a mixer and scaler video data of a back layer, a middle layer and afront layer, wherein the superimposed back layer, middle layer and frontlayer video data are scaled and the output of the mixer and scaler iscommunicated to the mixer; decoding in at least one decoder at least oneof video and still pictures, wherein the output of the at least onedecoder is communicated to the mixer and scaler, decoding in at leasttwo simultaneously working decoders synchronized graphics or subtitles,the output of each decoder being selectively communicated to a selectedone of the mixer and the mixer and scaler, at least one of the twodecoders communicates respective input data for display; andcommunicating output data of a renderer for the non-synchronizedgraphics being used as a menu to the mixer.